Papers with NLG systems
EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent surge of interest in developing evaluation metrics based on pretrained large language models (LLMs) can better cope with lexical variation. |
| Approach: | They propose to replace computation-intensive transformers with lighter alternatives and employ linear and quadratic approximations for alignment algorithms on top of LLM representations. |
| Outcome: | The proposed approach replaces computation-intensive transformers with lighter alternatives and employs linear and quadratic approximations for alignment algorithms on top of LLM representations. |
Slot-consistent NLG for Task-oriented Dialogue Systems with Iterative Rectification Network (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to natural language generation are prone to errors, such as neglecting input slot values and generating redundant slot values. |
| Approach: | They propose an iterative rectification network to improve general NLG systems . they apply bootstrapping algorithms to sample training candidates and incorporate reward . |
| Outcome: | The proposed methods significantly reduce the slot error rate for strong baselines. |
Natural Language Generation by Hierarchical Decoding with Linguistic Patterns (N18-2)
Copied to clipboard
| Challenge: | a common and mostly adopted method is the rule-based (or template-based) method for natural language generation. |
| Approach: | They propose a hierarchical decoding NLG model based on linguistic patterns in different levels. |
| Outcome: | The proposed method outperforms the traditional one with a smaller model size. |
RankME: Reliable Human Ratings for Natural Language Generation (N18-2)
Copied to clipboard
| Challenge: | Existing studies have shown that human evaluation for natural language generation often suffers from inconsistent user ratings. |
| Approach: | They propose a rank-based magnitude estimation method which combines continuous scales and relative assessments to improve the reliability of human ratings. |
| Outcome: | The proposed method significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. |
RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text (2020.emnlp-demos)
Copied to clipboard
| Challenge: | Existing studies on how humans perceive machine-generated text are limited due to the prohibitive cost of running human evaluation studies. |
| Approach: | They propose a task to detect the boundary at which a text passage starts off human-written transitions to being machine-generated. |
| Outcome: | The proposed system evaluates machine-generated news articles on a wide range of domains. |
Evaluating the Evaluation of Diversity in Natural Language Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for controlling diversity by tuning a “decoding parameter” affect form but not meaning. |
| Approach: | They propose a framework that measures correlation between a diversity metric and a parameter that controls some aspect of diversity in generated text. |
| Outcome: | The proposed framework outperforms existing methods in estimating diversity . it shows that humans outperformed existing methods but affect form but not meaning . |
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors (2024.naacl-long)
Copied to clipboard
| Challenge: | Natural language generation models are used for many downstream applications involving interpersonal communication, such as text completion, "smart" reply suggestions, and chatbot assistants. |
| Approach: | They conduct five case studies that perturb identity-related language features in NLG inputs to examine their assumptions about fairness. |
| Outcome: | The findings highlight open challenges around what constitutes “fair” or “good” NLG system behaviors. |
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist (2023.acl-long)
Copied to clipboard
| Challenge: | a systematic review of automatic evaluation metrics for Natural Language Generation (NLG) shows that task-agnostic metrics have a weak correlation with human . |
| Approach: | They propose a framework to assess the effectiveness of automatic metrics in three NLG tasks . they propose task-agnostic and human-aligned metrics to be used for evaluation . |
| Outcome: | The proposed framework provides access to the evaluation tools for three NLG tasks. |
Refocusing on Relevance: Personalization in NLG (2021.emnlp-main)
Copied to clipboard
| Challenge: | a number of negative effects exist when NLG systems are not grounded to a specific input text. |
| Approach: | They argue that NLG systems should focus on making use of additional context . they argue that relevance should be thought of as a crucial tool for user-oriented text-generating tasks . |
| Outcome: | The proposed approach is more of the rule than the exception, the authors argue . they argue that value-sensitive design represents a crucial path forward . |
Towards a Better Metric for Evaluating Question Generation Systems (D18-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics based on n-gram similarity do not correlate well with human judgments . large datasets for document Question Answering (QA) have enabled the development of end-to-end supervised models . |
| Approach: | They propose a scoring function to capture answerability of questions . they also integrate existing similarity metrics into the scoring function . |
| Outcome: | The proposed scoring function improves human judgments on question answerability . the proposed scoring functions are made publicly available . |
Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Past studies have shown biases in natural language generation systems but there has been little work on evaluating the bias evaluation approaches. |
| Approach: | They propose a method for evaluating biases in natural language generation systems by paraphrasing syntactic prompts with different syntaktic structures and paraphrazing them to evaluate demographic bias. |
| Outcome: | The proposed method is more robust and shows that some syntactic structures prompt more toxic content while others could prompt less biased generation. |
Is Reference Necessary in the Evaluation of NLG Systems? When and Where? (2024.naacl-long)
Copied to clipboard
| Challenge: | Despite recent advances in reference-free metrics, it has not been well understood when and where they can be used as an alternative to reference-based metrics. |
| Approach: | They propose to use reference-free metrics to evaluate NLG systems . they find they have a higher correlation with human judgment and greater sensitivity to deficiencies in language quality . |
| Outcome: | The proposed metrics exhibit higher correlation with human judgment and greater sensitivity to deficiencies in language quality. |
Language Model Augmented Relevance Score (2021.acl-long)
Copied to clipboard
| Challenge: | Existing metrics that compare the candidate with the human reference do not consider the context, resulting in poor correlation with human judgements. |
| Approach: | They propose a language model-aware metric that augments the human reference while considering the context to provide evaluation scores that correlate highly with human judgements. |
| Outcome: | The proposed metric achieves higher correlation with human reference judgements and differentiates well-formed candidates from adversarial samples to a larger degree. |
Towards Pragmatic Production Strategies for Natural Language Generation Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using language to communicate successfully requires effort. |
| Approach: | They propose a conceptual framework for the design of natural language generation systems that follow efficient and effective production strategies to achieve complex communicative goals. |
| Outcome: | The proposed framework is applied to visually grounded referential games and abstractive text summarisation tasks with real-world applications. |
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that evaluating NLG systems using pairwise comparisons is expensive as the number of human annotations grows linearly with k. |
| Approach: | They propose a framework to efficiently identify the top-ranked system by actively choosing system pairs for comparison using dueling bandit algorithms. |
| Outcome: | The proposed framework reduces human annotations by 80% on 13 NLG evaluation datasets spanning 5 tasks . |
Contrastive Error Attribution for Finetuned Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for error tracing do not detect faithfulness errors in NLG datasets. |
| Approach: | They propose a framework to identify and remove low-quality training instances that lead to undesirable outputs. |
| Outcome: | The proposed method outperforms existing methods for detecting faithfulness errors in NLG datasets. |
IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation (2021.emnlp-main)
Copied to clipboard
Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, Pascale Fung
| Challenge: | Lack of publicly available NLG benchmarks for low-resource languages poses a challenge . authors show that IndoBART and IndoGPT achieve competitive performance on all tasks . |
| Approach: | They propose a benchmark to measure natural language generation progress in three low-resource languages of Indonesia . they use a corpus of pretraining datasets to build their models . |
| Outcome: | The proposed benchmark measures progress in Indonesian, Javanese, and Sundanese . the results highlight the importance of pretraining on closely related, localized languages . |
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to optimism that they can serve as reliable evaluators of natural language outputs. |
| Approach: | They propose to use factuality metrics to evaluate natural language outputs . they find they misestimate the factual accuracy of NLG systems . |
| Outcome: | The proposed metrics are inconsistent with each other and often misestimate the factual accuracy of NLG systems, causing biases against paraphrased outputs and outputs that draw upon faraway parts of the source documents. |